CONFIDENTIAL BUSINESS INFORMATION 
—T 
August 14, 2020 


Jeffrey L. Quandt 

Chief, VDD-D Division 

Office of Defects Investigation 

National Highway Traffic Safety Administration 
1200 New Jersey Avenue, S.E. 

Washington, D.C. 20590 


Re: PE20-010 — Response to Information Request (First Submission) 
Dear Mr. Quandt, 


This letter responds to your information request (IR) regarding Preliminary Evaluation PE20-010 (PE), 
dated June 23, 2020. Thank you for the opportunity to respond. 


As areminder and per our agreement on August 5, you granted our request to extend the deadline of this 
submission by one week (or August 14) from the deadline in the IR because the IR’s transmission to 
Tesla was delayed by approximately one week later than it was dated. In addition, given the volume of 
material requested in the IR, you granted our request to extend the deadline to respond to Questions 9, 
10, 14, and 15 by an additional two weeks, or August 28, if needed. 


The following answers and accompanying attachments respond to your questions in the IR. 


1. State, by model and model year, the number of subject and peer vehicles Tesla has 
manufactured for sale or lease in the United States. Separately, for each subject and [peer] 
vehicle manufactured to date by Tesla, state the following: 


VIN; 

Model; 

Model Year; 

Date of manufacture; 

Date warranty coverage commenced; 

The current number of eMMC program/erase (P/E) cycles (if the vehicle is currently 
equipped with an aftermarket eMMC that was not supplied by Tesla, so state and provide 
the last recorded P/E cycle count); 

The date the eMMC P/E cycle was recorded; and 

The state in the United States where the vehicle was originally sold or leased (or 
delivered for sale or lease). 
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Provide the table in Microsoft Access 2010, or a compatible format, entitled PRODUCTION 
DATA.” See Enclosure 1, Data Collection Disc, for a pre-formatted table that provides further 
details regarding this submission. 


Please see the Excel file in the accompanying folder marked for Question 1. 


2. State the number of each of the following, received by Tesla, or of which Tesla is otherwise 
aware, which relate to, or may relate to, the alleged defect in the subject and peer vehicles: 


Consumer complaints, including those from fleet operators; 

Field reports, including dealer field reports; 

Reports involving a crash, injury or fatality; 

Property damage claims; and 

Third-party arbitration proceedings where Tesla is or was a party to the arbitration; and 
Lawsuits, both pending and closed, in which Tesla is or was a defendant or co-defendant. 


“2AM 


For subparts “a” through “f,” state the total number of each item (e.g., consumer complaints, field 
reports, etc.) separately. Multiple incidents involving the same vehicle are to be counted 
separately. Multiple reports of the same incident are also to be counted separately (i.e., a 
consumer complaint and a field report involving the same incident in which a crash occurred are 
to be counted as a crash report, a field report and a consumer complaint). 


In addition, for items “c” through “f’ provide a summary description of the alleged problem and 
causal and contributing factors and Tesla’s assessment of the problem, with a summary of the 
significant underlying facts and evidence. For items “e” and “f’ identify the parties to the action, as 
well as the caption, court, docket number, and date on which the complaint or other document 
initiating the action was filed. 


Please see the Excel file in the accompanying folder marked for Question 2. 


3. Separately, for each item (complaint, report, claim, notice, or matter) within the scope of your 
response to Request No. 2, state the following information: 


a. Tesla’s file number or other identifier used; 

b. The category of the item, as identified in Request No. 2 (i.e., consumer complaint, field 
report, etc.); 

Cc. Vehicle owner or fleet name (and fleet contact person), street address, email address 


and telephone number; 

Vehicle’s VIN; 

Vehicle’s make, model and model year; 

Vehicle’s mileage at time of incident; 

Incident date; 

Report or claim date; 

Whether a crash is alleged; 

Whether property damage is alleged; 

Number of alleged injuries, if any; 

Number of alleged fatalities, if any; 

Whether the MCU was replaced by Tesla; 

The number of eMMC P/E cycles at the report or claim date; 
Tesla’s assessment of the applicable subcategory of the alleged defect; and 
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p. A short summary of the symptoms alleged. 


Provide this information in Microsoft Access 2010, or a compatible format, entitled “REQUEST 
NUMBER TWO DATA.” See Enclosure 1, Data Collection Disc, for a pre-formatted table that 
provides further details regarding this submission. 


Please see the Excel file in the accompanying folder marked for Question 3. 


4. Produce copies of all documents related to each item within the scope of Request No. 2 Organize 
the documents separately by category (i.e., consumer complaints, field reports, etc.) and describe 
the method Tesla used for organizing the documents. Describe in detail the search methods and 
search criteria used by Tesla to identify the items in response to Request No. 2. 


Please see the attachments in the accompanying folders marked for Question 4, organized by category. 


The format and extent of information produced for consumer complaints and field reports were organized 
into the Excel templates that we discussed and agreed upon with you on August 5. In addition, and as 
agreed on August 5, if a consumer complaint or field report included an attachment (e.g., a photo or video 
clip), then we marked the last column in each template affirmatively (Attachment (Y/N)) and produced the 
attachment as a separate file whose name corresponded with the respective line item in the template. To 
the extent that we were able to produce a copy of the actual consumer complaint or field report, we also 


marked the last column affirmatively and produced it in the same fashion as a photo or video clip. 


Of the consumer complaints captured in the template, we identified 57 attachments for production. Of the 
field reports captured in the template, we identified 1,699 attachments for production. Please note that for 
one of the sources of consumer complaints (Customer Connect, a new complaint logging platform 
introduced in late 2019), we are unable to provide any attachment files with this submission. The reason 
is because all attachment-related data recorded in Customer Connect is not part of the actual database 
warehouse. We will work with the application management team and provide an update with our second 
submission as to the availability and extent of responsive attachments from the Customer Connect 


source. 


Below are the details on the search methods and search criteria we used to identify responsive items for 


Question 2. 


For subsection 2(a), we queried all sources of consumer complaints logged by Tesla and further filtered 
down to those complaints that are related to the PE’s definition of “alleged defect.” A list of symptoms that 
represent any one of the three prongs of alleged defect definition were used in the search process. The 


search was further refined to identify truly responsive complaints by looking for application of any of the 
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applicable correction codes (“labor operation numbers”) representing repair or replacement of MCUs, 
consumption of any one of the MCU parts, or a record of any one of the applicable trouble codes 


(“Toolbox articles”). Further below we have explained the list of search criteria in detail. 


For subsection 2(b), we queried all field reports (Toolbox sessions that were escalated to technical 
specialists for assistance with diagnosis) that are related to the PE’s definition of “alleged defect.” The 
search was refined further by filtering for records related to applicable symptoms and labor operation 
numbers. We also reviewed Watchtower field reports to add any responsive ones to the final set of field 
reports. Watchtower is a web platform through which Field Technical Specialists (FTSs) create field 
reports for vehicle repairs of interest, so that they may be shared with other FTSs. More specifically, each 
FTS supports a number of service locations and creates field reports at his or her discretion, but the 
reports are generally created for new, unusual or otherwise interesting repairs that the FTS believes 
would be of value to share with other FTSs. 


Below is the list of search criteria we used to identify responsive items for subsection 2(a) and 2(b). 


e Symptoms: Symptoms are pre-defined descriptions of perceived vehicle behavior. In fielding a 
customer complaint or documenting a repair order or a field report, a Tesla team member selects 
a symptom that is most representative of the issue at hand. To define a set of symptoms that 
would be responsive to Question 2, we first narrowed the symptom list down to those that 
contained at least one of the following key words — “MCU,” “touchscreen,” “display,” “blank,” 
“camera.” We reviewed the resulting list of symptoms for responsiveness to any one of the three 
prongs of NHTSA’s definition of “alleged defect.” The result of this review is a set of symptoms we 


used to find responsive customer complaints, field reports and repair data. 


e Correction codes: We identified a list of correction codes (“labor operation numbers”) that 
represent replacement or repair of MCUs. We also included those that represent MCU upgrades 
because a customer may opt to upgrade her MCU instead of repairing or replacing a 


malfunctioning unit. 


e Parts: To assist with identifying items responsive to the alleged defect, we identified a list of 
replacement MCU parts, an MCU subcomponent part that may be used to address the alleged 
defect, and MCU upgrade parts because a customer may opt to upgrade her MCU instead of 


repairing or replacing a malfunctioning unit. 
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e Trouble Codes: To assist with diagnosis of a given issue, Tesla Technicians may create a 
“Toolbox session” to document findings and, if necessary, “escalate” to technical specialists for 
assistance with diagnosis. In a Toolbox session, technicians may mark certain “Toolbox articles,” 
which may contain diagnostic information, as a confirmed solution to the particular issue they are 
working through in the session. Wherever available for repair data responsive to the PE request, 
we identified the “trouble code” for a given repair job as the Toolbox article number that a 
technician identified as the solution for that repair job. Accordingly, we identified a set of Toolbox 
articles and article numbers that are responsive to NHTSA’s PE request, and we used those 
articles numbers (or “trouble codes”) to identify potentially responsive repairs by looking for jobs 


where a technician marked one of those articles as the solution to the job. 


For subsection 2(c), we interpreted "report” as EWR D&l reports because the term was otherwise 
undefined and because the other subsections in Question 2 were already so comprehensive. After a 
review of prior EWR D&l reports, we did not find any responsive incidents, such as reports that were 
marked for back over prevention, rearview camera or MCU malfunction. Moreover, the 
comprehensiveness of the other subsections in Question 2 capture all other documents that could be 
remotely responsive under the broadest definition of “report” in 2(c). For this reason, we concluded that 


no reports were responsive for subsection 2(c). 


For subsection 2(d), the product liability team within the Tesla Legal Department tracks all demand letters 
from customers, insurance carriers or counsel wherein there is an allegation that a vehicle defect caused 
or contributed to a crash resulting in property damage, personal injury or death. In order to respond to 
NHTSA’s request, the team reviewed demand letters relating to vehicles within the identified scope and 


found none were responsive. 


For subsection 2(e), one lawyer within Tesla handles all consumer arbitrations relating to alleged 
warranty issues (/.e., “lemon law” claims) and maintains a database of such arbitrations with information 
including the customer, VIN/vehicle information, venue, a brief identification of the alleged failing 
components (e.g., MCU), and the results of the proceeding. The potentially responsive arbitrations were 


reviewed to ensure responsiveness and are identified and provided. 


Finally, for subsection 2(f), Tesla has not received any lawsuits alleging property damage or injury/death 
within the scope of the affected vehicle and alleged defect requested by NHTSA. Tesla has received 
lawsuits alleging warranty defects within the defined scope (/.e., “lemon law’ cases). The litigation team 
within the Legal Department maintains a database of such litigation with information including the case 


name, docket information, VIN/vehicle information, venue, open/close dates, and brief identification of the 
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alleged failing components, among other things. The potentially responsive lawsuits were reviewed to 
ensure responsiveness and are identified and provided. 


5. State, by model and model year, a total count for all of the following categories of claims, 
collectively, that have been paid by Tesla to date that relate to, or may relate to, the alleged 
defect in the subject and peer vehicles: warranty claims, extended warranty claims, claims for 
goodwill services that were provided; and field, zone, or similar adjustments and reimbursements. 


Separately, for each such claim, state the following information: 


a. Tesla’s claim number; 

b. Vehicle owner or fleet name (and fleet contact person), street address, email address 
and telephone number; 

C. VIN; 

d. Repair date; 

e. Vehicle mileage at time of repair; 

f Repairing dealer’s or facility’s name, telephone number, city and state or ZIP code; 

g. Labor operation number(s); 

h. Problem code(s); 

i Diagnostic trouble code(s); 

if Replacement part number(s) and description(s); 

k. The number of eMMC P/E cycles when repaired; 

I Concern stated by customer; 

m. Cause as stated on the repair order; 

n. Correction as stated on the repair order; and 

O. Additional comments, if any, by dealer/technician relating to claim and/or repair. 


Provide this information in Microsoft Access 2010, or a compatible format, entitled “WARRANTY 
DATA.” See Enclosure 1, Data Collection Disc, for a pre-formatted table that provides further 
details regarding this submission. 


Please see the Excel file in the accompanying folder marked for Question 5. 


6. Describe in detail the search methods and search criteria used by Tesla to identify the claims in 
response to Request No. 5, including the labor operations, problem codes, diagnostic trouble 
codes, part numbers and any other pertinent parameters used. 


Provide a list of all labor operations, labor operation descriptions, problem codes, and problem 
code descriptions, diagnostic trouble codes and diagnostic trouble code descriptions applicable to 
the alleged defect in the subject vehicles. State whether the diagnostic trouble codes are 
automatically reported to the warranty database electronically or manually entered into the 
warranty database by a claims administrator. 


State, by make and model year, the terms of the new warranty coverage offered by Tesla on the 
subject vehicles (i.e., the number of months and mileage for which coverage is provided and the 
vehicle systems that are covered). Describe any extended warranty coverage option(s) that Tesla 
offered for the subject vehicles and state by option, model, and model year, the number of 
vehicles that are covered under each such extended warranty. 


Please find below details on the search methods and criteria we used to identify the repair claims. 
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We queried all repair jobs tied to a list of symptoms that represent any one of the three prongs of 
NHTSA’s definition of “alleged defect.” Below is a summary of the method we used to identify the set of 


applicable symptoms: 


e Symptoms: Symptoms are pre-defined descriptions of perceived vehicle behavior. In fielding a 
customer complaint or documenting a repair order or a field report, a Tesla team member selects 
a symptom that is most representative of the issue at hand. To define a set of symptoms 
responsive to NHTSA’s PE request, we first narrowed the symptom list down to those that 
contained at least one of the following key words — “MCU”, “touchscreen”, “display”, “blank”, 
“camera”. We reviewed the resulting list of symptoms for responsiveness to any one of the three 
prongs of NHTSA’s definition of “alleged defect.” The result of this review is a set of symptoms we 
used to find responsive customer complaints, field reports and repair data. 


The resulting data was further refined to identify responsive repairs by looking for application of any of the 
applicable correction codes (“labor operation numbers”) representing repair or replacement of MCUs, 
consumption of any one of the MCU parts, or a record of any one of the applicable trouble codes 
(“Toolbox articles”). 


e Correction codes: We identified a list of correction codes (“labor operation numbers”) that 
represent replacement or repair of MCUs. We also included those that represent MCU upgrades 
because a customer may opt to upgrade her MCU instead of repairing or replacing a 


malfunctioning unit. 


e Parts: To assist with identifying items responsive to the alleged defect, we identified a list of 
replacement MCU parts, an MCU subcomponent part that may be used to address the alleged 
defect, as well as MCU upgrade parts because a customer may opt to upgrade her MCU instead 


of repairing or replacing a malfunctioning unit. 


e Problem Codes: “Toolbox articles” represent content created by technicians in the field that 
describe conditions present on vehicles and may contain diagnostic or procedural information. 
This collaboratively authored content is moderated by the Tesla engineering team but is largely 
generated by the field. As such, it may contain inaccurate, irrelevant, or misleading information. 
Technicians have the option to mark Toolbox articles as relating to the condition they believe to 
be present on the vehicle. Wherever available for repair data responsive to the PE request, we 
identified the “problem code’ for a given repair job as non-FAQ, acceptable endpoint Toolbox 


article number that a technician identified as the condition present for that repair job. Accordingly, 
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we identified a set of Toolbox articles and article numbers that are responsive to NHTSA’s PE 


request, and we used those article numbers (or “problem codes”) to identify potentially responsive 


repairs by looking for jobs where a technician marked one of those articles as describing the 


condition present on the job. 


e Diagnostic Trouble Codes: Diagnostic trouble codes are not automatically reported to a warranty 


database electronically. It is up to the discretion of the technician performing the repair whether 


they mention or do not mention present diagnostic codes in the Cause Narrative. 


Please see the Excel file in the accompanying folder marked for Question 6 for a list of applicable labor, 


problem, and diagnostic codes. 


Please also see the separate Excel file in the accompanying folder marked for Question 6 for new 
warranty coverage by years and mileage, including Basic Vehicle Limited Warranty (non-powertrain), 
Battery and Drive Unit Limited Warranty (powertrain), and Supplemental Restraint System Limited 

Warranty (safety and restraints, airbags). The same excel sheet includes a tab for extended service 


agreement offers. 


7. State, by model and model year, a total count for all customer pay repairs that relate to, or may 
relate to, the alleged defect in the subject and peer vehicles (to the extent the requested data is 


available). 


Separately, for each such claim, state the following information: 


a. Tesla’s claim number; 

b. Vehicle owner or fleet name (and fleet contact person), street address, email address 
and telephone number; 

C. VIN; 

d. Repair date; 

e. Vehicle mileage at time of repair; 

f Repairing dealer’s or facility's name, telephone number, city and state or ZIP code; 

g. Labor operation number(s); 

h. Problem code(s); 

i Diagnostic trouble code(s); 

j. Replacement part number(s) and description(s); 

k. The number of eMMC P/E cycles when repaired; 

I Concern stated by customer; 

m. Cause as stated on the repair order; 

n. Correction as stated on the repair order; and 

O. Additional comments, if any, by dealer/technician relating to claim and/or repair. 


Provide this information in Microsoft Access 2010, or a compatible format, entitled “NON- 
WARRANTY DATA.” See Enclosure 1, Data Collection Disc, for a pre-formatted table that 
provides further details regarding this submission. 
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Please see the Excel file in the accompanying folder marked for Question 7. 


8. Produce copies of all service, warranty, and other documents that relate to, or may relate to, the 
alleged defect in the subject and peer vehicles, that Tesla has issued to any dealers, regional or 
zone offices, field offices, fleet purchasers, or other entities. This includes, but is not limited to, 
bulletins, advisories, informational documents, training documents, or other documents or 
communications, with the exception of standard shop manuals. Also include the latest draft copy 
of any communication that Tesla is planning to issue within the next 120 days. 


Please see the attachments in the accompanying folder marked for Question #8, which are organized by 
Announcements, Hub resources, Confluence pages, Toolbox articles, and bulletins. These resources 
capture the extent of documentation that this question requests for production, and a general description 


of each is provided below. 


Announcements are prepared by Service Operations with support from subject matter experts, including 
Service Engineering, and are deployed to all service centers — in this case, all North American service 
centers — to periodically update the centers on helpful information, trends, or general clarifications. 
Announcements typically include links to Hub announcements, which is where the topical content of the 
Announcement is hosted. Hub announcements, in turn, often include links to other in-depth resources 
including Hub articles, Toolbox articles, Confluence pages and bulletin documents. Hub articles provide 
basic topic overviews, guidelines for proper service action, and reference material to assist frontline staff 


with fielding questions from customers. 


Confluence pages and Toolbox articles are resources for service technicians to describe, understand, or 
diagnose various repair conditions. “Toolbox articles” represent content created by technicians in the field 
to describe conditions present on vehicles and may contain diagnostic or procedural information. This 
collaboratively authored content is moderated by the Tesla engineering team but is largely generated by 
the field. As such, it may contain inaccurate, irrelevant, or misleading information. As further discussed in 
Question 6, the nature of Toolbox articles can also meet a loose definition for problem codes. 
Confluences pages and Toolbox articles are not communicated to service technicians like an 
announcement. Technicians are not notified when other technicians or Service Engineering create new 
articles, although occasionally Service Engineering will reference them in announcements to service 
centers. Confluence pages and Toolbox articles are more akin to an internal searchable database, which 
explains why they do not copy well for hard- or soft-copy production. However, like a searchable 
database, technicians can follow various articles of interest or relevance and receive notification when 
Service Engineering updates them. A Toolbox article and the content contained within can grow obsolete 


over time. An article will be marked as obsolete if the content or condition is no longer relevant to the vast 
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majority of the fleet, if a separate article or content has superseded it, or if it pertained to a calendar date 
that has since elapsed. When an article is marked as obsolete, it is no longer available to service 


technicians in the field, and the content is no longer actively maintained. 


Finally, service bulletins provide instructions to trained service technicians on how to perform a specific 
repair or address a noted condition or possible customer concern. A subset of this category are service 
internal documents, which, as the name suggests, are kept internal to Tesla personnel, including service 
center staff, because they may discuss proprietary information, contain how-to procedures for revenue- 
generating upgrades, or share specific safety precautions or training necessary to perform a particular 
repair. The only bulletin relevant for purposes of answering this question is a service internal document, 


and it was designated as such because it discusses proprietary information about the MCU. 


9. Describe all assessments, analyses, tests, test results, studies, surveys, simulations, 
investigations, inquiries and/or evaluations (collectively, “actions’) that relate to, or may relate to, 
the alleged defect in the subject and peer vehicles that have been conducted, are being 
conducted, are planned, or are being planned by, or for, Tesla. For each such action, provide the 
following information: 


Action title or identifier; 

The actual or planned start date; 

The actual or expected end date; 

Brief summary of the subject and objective of the action; 

Engineering group(s)/supplier(s) responsible for designing and for conducting the action; 
and 

A brief summary of the findings and/or conclusions resulting from the action. 


2Q295 


Pima 


For each action identified, provide copies of all documents related to the action, regardless of 
whether the documents are in interim, draft, or final form. Organize the documents chronologically 
by action. 


As per our prior agreement on August 5, we will respond to this question in a second, follow-up 


submission by August 28. 


10. Describe all modifications or changes made by, or on behalf of, Tesla in the design or 
programming of the subject component, from the start of production to date, which relate to, or 
may relate to, the alleged defect in the subject and peer vehicles. Include all over-the-air updates 
to vehicle software/firmware related to the alleged defect, include changes in default operating 
modes and changes in memory use by the subject eMMC. For each such modification or change, 
provide the following information: 


a. The date or approximate date on which the modification or change was incorporated into 
vehicle production; 

b. A detailed description of the modification or change; 

C. The reason(s) for the modification or change; 

d. The part number(s) (service and engineering) of the original component; 

ée. The part number(s) (service and engineering) of the modified equipment; 
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f Whether the original unmodified component was withdrawn from production and/or sale, 
and if so, when; 

When the modified component was made available as a service component; and 
Whether the modified component can be interchanged with earlier production 
components. 


7a 


Also, provide the above information for any modification or change that Tesla is aware of which 
may be incorporated into vehicle production within the next 120 days. 


As per our prior agreement on August 5, we will respond to this question in a second, follow-up 


submission by August 28. 


17. 


State the number of subject components that Tesla has sold that may be used in the subject and 
peer vehicles by component name, part number (both service and engineering / production), 
eMMC storage capacity, model and model year of the vehicle in which it is used and month/year 
of sale (including the cut-off date for sales, if applicable). 


For each component part number, provide the supplier's name, address, and appropriate point of 
contact (name, title, and telephone number). Also identify by make, mode and model year, any 
other vehicles of which Tesla is aware that contain the identical component, whether installed in 


production or in service, and state the applicable dates of production or service usage. 


In addition, state whether Tesla is able to identify subject vehicles equipped with eMMC storage 
devices installed by third-party repair facilities. If so, provide a description of how the vehicles are 
identified and provide an Excel file listing all such vehicles by VIN, date and mileage the eMMC 


appears to have been replaced. 


As of August 13, 2020, below are the subject components that Tesla has sold: 


Number 
euniee: eMMC | Vehicle | Model | Month/Year | S014 
Component Part Number(s) | Supplier storage Model Véar of Sale (as of 
Name capacity July 

2020) 

1004777-00-C 

1004777-00-E 

1004777-00-F 

1004777-00-G 

1004777-00-H 
Tegra MCU 1004777-00-J Hynix 2012- June 2012 — 
(Production) 1004777-00-K | Tesla. Inc. | gap ModelS | 5015 | Dec 2015 mae 

1004777-00-L 

1004777-00-M 

1004777-00-N 

1004777-03-C 

1004777-03-E 
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1004777-03-F 
1004777-03-G 
1004777-06-E 
1004777-06-F 
1004777-16-G 
1004777-16-I 

1004777-16-J 

1004777-20-B 


Tegra MCU 
(Production) 


1045006-00-A 
1045006-00-B 
1045006-00-C 
1045006-00-D 
1045006-00-E 
1045006-02-E 
1045006-03-A 


Tesla, Inc. 


Hynix 
8GB 


Model S,X 


2015- 
2018 


Dec 2015 — 
March 2018 


95,963 


Tegra MCU 
(Service 
replacements) 


1004777-00-B 
1004777-00-C 
1004777-00-E 
1004777-00-F 
1004777-00-G 
1004777-00-H 
1004777-00-J 
1004777-00-K 
1004777-00-L 
1004777-00-M 
1004777-01-C 
1004777-02-C 
1004777-03-C 
1004777-03-E 
1004777-03-F 
1004777-03-G 
1004777-04-C 
1004777-06-E 
1004777-15-N 
1004777-16-| 
1004777-20-C 
1004777-40-L 
1004777-41-M 
1004780-00-C 
1004780-00-E 
1004780-00-F 
1004780-00-G 
1004780-00-J 
1004780-00-K 
1004780-00-L 
1004780-00-M 
1004780-03-C 
1004780-03-E 
1004780-03-F 
1004780-03-G 
1004780-06-F 
1004780-40-A 
1004780-41-A 
1004780-88-C 
1045006-00-A 
1045006-00-B 
1045006-00-C 
1045006-00-D 
1045006-00-E 


Tesla, Inc. 


Hynix 
8GB 


Model S,X 


2012- 
2018 


Sept 2012 


25,487 
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1045006-00-H 

1045006-00-J 

1045006-02-F 

1045006-03-A 
1045006-03-H 
1045006-04-E 
1045006-S0-J 

1098010-00-A 
1098010-00-B 
1098010-00-D 
1098010-00-F 
1098010-00-G 
1098010-88-B 
1098010-88-D 
1098010-88-F 
1098050-00-F 
1452455-00-B 
1452455-00-C 
1452455-02-C 
1458829-00-A 
1458829-00-B 


Tegra MCU 
(Service 
replacements) 


1098010-90-B 
1098010-99-H 
1098050-99-H 


Tesla, Inc. 


Micron 
64GB 


Model S,X 


2012- 
2018 


May 2020 


3,396 


Please note that, starting July 27, 2020, Tesla made the Tegra Visual Control Module (VCM) available for 
replacement with a 64GB Micron eMMC, avoiding a full MCU replacement (see Question 12 for details on 


VCM and MCU architecture). These replacements are not accounted for in the table above. 


Regarding replacements by third-party repair facilities, we can detect vehicles with unknown eMMC 


names or serial numbers via our fleet-wide logging and can infer the list of vehicles with a replacement in 
a third-party repair facility of an unapproved eMMC device. We have no record of the date or mileage at 
the time of replacement. The vehicles detected as of August 8, 2020, can be found in the attachment 


“Vehicles with unknown eMMC devices.csv” in the accompanying folder marked for Question 11. 
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12. Provide a diagram of the architecture of in-vehicle electronics, including the subject component, 
other electronic control units, communication networks, and external connectivity ports (cellular, 
Wi-Fi, Bluetooth). 


The center display, commonly referred to as Media Control Unit (MCU), is the primary touchscreen 
interface for Tesla Model S/X infotainment systems, located at the center of the instrument panel, 
between the driver and passenger seats. The MCU is directly connected to the center display 
touchscreen, which contains the display panel and touch panel. In all vehicles, from start of production 
until March 2018, the MCU consists of the following main components: 


e First, the Visual Computing Module (VCM), or processor module, which contains the NVidia 
Tegra3 System-On-Chip (SoC), and associated peripherals, including volatile memory (DDR2), 
persistent storage (eMMC), ethernet/JTAG ports, and others functions directly linked to the SoC. 
The VCM, illustrated below, is on its own daughterboard and is connected to the motherboard via 


a high-density connector. 


JTAG/ 
Ethernet 
bce Connector 
DDR: (opt) 
SLC NAND/ Memory 


NOR Flash 
Memory 


SPI- 
Ethernet 
Bridge (opt) 


e Second, the motherboard, or Printed Circuit Board Assembly (PCBA), which contains other 
peripherals: connectivity module, WiFi / Bluetooth module (referenced as ‘Parrot’), ethernet 
switch, audio digital signal processor, display field-programmable gate array (FPGA), and the 
Gateway processor (high-reliability component interfacing with the rest of the vehicle over CAN). 


The Gateway processor is a high-reliability control unit, based on the NXP MPC5668G (part number of 
the Gateway ECU). It communicates with the center display and cluster display via ethernet, and to the 
vehicle Electronic Control Units (ECU) via CAN. 


The instrument cluster is an independent hardware module responsible to display information to the driver 
on a separate display, directly behind the steering wheel. It communicates with the center display via 
ethernet, based on the same VCM with the NVidia Tegra2 SoC. The cluster display is capable of 
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operating independently from the center display, and it communicates to the rest of the vehicle via the 
Gateway over ethernet. 


The MCU architecture is further illustrated in the diagram and photos below: 
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Bottom View: 
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13. Provide the following information regarding subject eMMC memory devices: 
a. Supplier; 


Nvidia Corporation supplies the Visual Computing Module (VCM), which utilizes an Embedded Multi- 
Media Card (eMMC) memory device supplied by SK hynix Inc (commonly referred to as “Hynix’). 


b. Data sheets; 


There are two variants of the Hynix eMMC on the Nvidia VCM: 


e =§=26nm variant: 

o Part number: H26M42001FMR 
SK Hynix 26nm 32Gb e-NAND Product Family Datasheet: Refer to attachment 
Hynix_H26M42001FMR_F26_32Gb_eNAND_Family_rev_2.0.pdf 


(e) 
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e = §=20nm variant: 
o Part number: H26M42003GMR 
o SK Hynix 20nm 32Gb e-NAND Product Family Datasheet: Refer to attachment 20nm 
eMMC Datasheet_Rev1.4_20130916.pdf 


For detailed information about each variant, please see the attachments in the accompanying folder 


marked for Question 13. 

Cc. Storage capacity; 
The storage capacity for all eMMC of the Tegra VCM at production was 8 GB. As of May 2020, Tesla 
remanufacturing began producing spare parts which incorporate a Micron eMMC with a 64GB storage 
capacity. 

d. Describe the layout / configuration of the eMMC flash memory; 


The eMMC is partitioned into six distinct regions with the following layout: 


Tegra eMMC storage (8 GB) 


mmcbIkObootO 
Unused (8 MB) 


mmcblikOboot1 
Backup data (8 MB) 
mmcbikOp1 
Operating System read-only data 
Bank A 
(1 GB) 
mmcblikOp2 
Operating System read-only data 


Bank B 
(1 GB) 


mmcblik0p3 
Operating thy variable data 
(128 MB) 


mmcbik0p4 
User Data 
(5.2 GB) 
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The Operating System read-only regions (mmcblkOp1 and mmcblkOp2) are implemented as compressed 
“SquashFS’ file systems. There are two banks to support A/B style firmware updates. The Operating 


System variable and user data regions are implemented as Linux journaled “ext3” file systems. 


e. Describe error correction and wear leveling strategies that may affect device durability; 


The subject eMMC devices are compliant with the JEDEC eMMC standard JESD84-A441, which 
necessitates that the controller within the eMMC device is responsible for presenting flash blocks as 
logical units to the host Operating System and is responsible for wear leveling and bad block 


management. 


The inner workings of Hynix’s wear leveling strategies are not known to Tesla. The essential function of 
the wear leveling is to spread Program/Erase (P/E) cycles evenly throughout the pool of physical flash 
blocks. 


The eMMC device is capable of reporting health information to indicate how effective the wear leveling is 
functioning. It reports the average number of P/E cycles for each block across the device, as well as the 
minimum and maximum counts seen. An effective wear leveling algorithm will generally keep these 
numbers relatively close to each other. Our internal examinations of these numbers have found that all 


three tend to be close to each other after significant usage. 


Additionally, the controller within the eMMC has the capability to detect certain failure modes (e.g., write 
failure for a given physical block) where it can mark a block as “bad” and replace it with a spare/reserved 
block without intervention from the host Operating System. The effectiveness of this error correction for 
increasing durability is a function of (1) fault detection coverage for various failure modes, and (2) the 
number of spare/reserved blocks available. 


f Rated number of P/E cycles; 


The Hynix eMMC utilized by the Nvidia VCM are rated for 3,000 Program/Erase cycles for each block of 
NAND flash within the eMMC. 


g. Nominal daily P/E cycles; 


For vehicles running firmware version 2020.20 or newer—release 2020.20 is the most widely installed 


across the fleet, currently at 93%—each block undergoes an average increase of 0.7 P/E cycles per day. 
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h. Describe all OTA updates affecting P/E cycle use rate that have been released by Tesla; 


The firmware releases available over-the-air (OTA) that affect P/E cycles are listed below. All changes 


listed in a particular release are included in subsequent releases, including minor releases of the same 


major version (é.g., 2019.24.2 includes changes listed in 2019.24). 


Firmware release Date Changes affecting P/E cycle use 
2019.24 July 2019 Remove high frequency system log messages 
Reduce significant log writes caused by faulty USB devices 
2019.28 August 2019 
plugged into front console hub 
Increase journal commit interval time for writeable file system to 
2019.36 Nov-Dec 2019 St 
reduce Write Amplification Factor 
Suppress informational system log messages when user not 
2019.32.12.7 Nov 2019 
present in vehicle 
2020.16 May 2020 Reduce settings database write frequency. 
Sept-Oct 2020 Add initial support for reducing eMMC usage as a cache for 
2020.32 (scheduled) 
(scheduled) various services 
i Factors that influence daily storage use (with estimated upper bound); and 


The factors influencing daily storage use are dominated by vehicle usage patterns: 


e Daily drive time; 


e Daily charge time; and 


e Use of Internet music streaming while in vehicle. 


i Expected service life in years at: (1) nominal daily P/E cycle use; and (2) 95" percentile 
daily P/E cycle use. 


At the nominal daily P/E cycle use rate of 0.7 per block, it would take between 11 — 12 years 


to accumulate an average of 3,000 P/E cycles per block in the device. 


At the 95" percentile daily P/E cycle use rate of 1.5 per block it would take 5 — 6 years to accumulate an 


average of 3,000 P/E cycles per block in the device. 


14. 
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Provide the following information about eMMC memory wear-out in the subject components: 


a. 


aac 


® 


Pi | 


Dea 


Describe operations that use flash memory (include description of the original design and 
all changes implemented to date); 

Tesla’s assessment of the root cause of MCU failures in the subject vehicles; 

Symptoms of an eMMC card that is nearing end-of-life memory wear; 

Vehicle control functions, audible alerts, and safety telltale displays that are lost when 
MCU fails (black screen); 

Driver options for performing control functions by alternate means (e.g., steering wheel 
controls), if applicable; 

Driver options for displaying system status and/or telltale symbols on alternate screens 
(e.g., dashboard user interface); 

Audible warning chimes; 

Safety restraint telltales; 

Effects on exterior lighting control; and 

Description of climate control function after MCU failure to provide defogging / defrosting 
functions required for visibility. 


As per our prior agreement on August 5, we will respond to this question in a second, follow-up 


submission by August 28. 


15. 


Furnish Tesla’s assessment of the alleged defect in the subject vehicle, including: 


2295 


Pi | 


The causal or contributory factor(s); 

The failure mechanism(s); 

The failure mode(s); 

The risk to motor vehicle safety that it poses; and 

What warnings, if any, the operator and the other persons both inside and outside the 
vehicle would have that the alleged defect was occurring or the subject component was 
malfunctioning; and 

The NHTSA complaints sent to Tesla for review. 


As per our prior agreement on August 5, we will respond to this question in a second, follow-up 


submission by August 28. 


Thank you for your consideration. If you have any questions, please contact me at erwilliams@tesla.com. 


Sincerely, 
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Eric C. Williams 


Associate General Counsel, Regulatory 


